Skip to content

fix(s3): preserve datetime64[us] resolution for timestamps outside na… - #3359

Open
bujjibabukatta wants to merge 5 commits into
aws:mainfrom
bujjibabukatta:fix/#3357
Open

fix(s3): preserve datetime64[us] resolution for timestamps outside na…#3359
bujjibabukatta wants to merge 5 commits into
aws:mainfrom
bujjibabukatta:fix/#3357

Conversation

@bujjibabukatta

Copy link
Copy Markdown
Contributor

Closes #3357

Problem

pandas 3.0 defaults timestamps to datetime64[us] resolution. When writing
parquet files containing timestamps outside the nanosecond range (before 1677
or after 2262), data was silently corrupted on read-back.

Root cause: the write defaults flavor="spark" and version="1.0" in
s3/_write_parquet.py force all timestamps to INT96 physical storage
(nanosecond-only). Coercing datetime64[us] values outside the nanosecond
range into INT96 overflows, producing garbage timestamps on read.

Changes

awswrangler/s3/_write_parquet.py

  • coerce_timestamps: "ms""us" — prevents microsecond precision loss
  • flavor: "spark"None — removes forced INT96 physical storage
  • version: "1.0""2.6" — uses typed timestamps instead of INT96

awswrangler/_data_types.py

  • pyarrow2pandas_defaults: add coerce_temporal_nanoseconds=False — prevents
    pyarrow from casting timestamp[us] back to datetime64[ns] on read
  • athena2pyarrow: map all pandas datetime resolutions (ns/us/ms/s) to the
    correct pyarrow timestamp unit; default to us for pandas 3.0+
  • Fix pre-existing bug: dtype in ("binary" or "varbinary")
    dtype in ("binary", "varbinary")"binary" or "varbinary" evaluated
    to "binary" only, causing varbinary to raise UnsupportedType

Verification

import pandas as pd
import awswrangler as wr

df = pd.DataFrame({
    "ts": pd.to_datetime([
        "1000-01-01 12:00:00",  # outside ns range
        "1677-10-01 12:00:00",  # inside ns range
        "3000-01-01 12:00:00",  # outside ns range
    ])
})
# Before: read-back returned corrupted timestamps for rows 0 and 2
# After:  all rows round-trip correctly as datetime64[us]

@bujjibabukatta

Copy link
Copy Markdown
Contributor Author

Hi @kukushking could you please review and approve pull request?

@kukushking kukushking left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, unfortunately this change breaks backwards-compatibility beyond fixing the bug (timestamp overflow) it intends to fix by changing three things at once, each having their own consequences:

  1. coerce_timestamps: "ms" → "us" is the load-bearing change for the bug fix, and the safest of the three. Fine - could land alone.
  2. version: "1.0" → "2.6" - necessary because typed timestamps need 2.x logical types. It is defensible, however we need to document the EMR/legacy-Spark compatibility hit and consider gating with a deprecation path: keep 1.0 default for one release, emit a DeprecationWarning, flip in the next major.
  3. flavor: "spark" → None is not required Keep flavor="spark" (or document explicitly that this is a separate, intentional behavior change about column-name sanitization).

@bujjibabukatta

bujjibabukatta commented Jun 17, 2026

Copy link
Copy Markdown
Contributor Author

Hi @kukushking Thanks for the review! Reverted flavor back to "spark" and version back to "1.0", keeping only the coerce_timestamps "ms" → "us" change as the actual bug fix. Let me know if this looks good!

@bujjibabukatta

Copy link
Copy Markdown
Contributor Author

Hi @kukushking I have updated code based on your comments. Please approve pull request?

@bujjibabukatta

bujjibabukatta commented Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

Hi @kukushking, @robert-schmidtke could you please review and approve pull request?

@robert-schmidtke robert-schmidtke left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honestly I am not sure about the fix. Just using microseconds instead of nanoseconds unconditionally feels like it could be breaking to environments that use pandas 2.x.

As I am merely a reporter of the bug I am not really in a position to give a go/no go on this PR. At work I would suggest to add some test cases (maybe across different pandas versions even) to ensure the timestamps are preserved correctly.

Comment thread awswrangler/_data_types.py Outdated
return pa.timestamp(unit="s")
else:
return pa.timestamp(unit="ns")
return pa.timestamp(unit="us") # pandas 3.0 default is us not ns

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this return nanoseconds or microseconds based on the actual pandas version installed? Otherwise this would break with pandas 2.x, right?

@bujjibabukatta

bujjibabukatta commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Hi @kukushking @robert-schmidtke addressed both: kept flavor="spark" and
version="1.0" unchanged, but found flavor="spark" alone was still
reproducing the overflow bug regardless of version. Fixed it by explicitly
setting use_deprecated_int96_timestamps=False, which resolves the overflow
without changing flavor or version at all — verified locally against the
original repro.

Also fixed the pandas-version assumption in _data_types.py: it now checks
the installed pandas version instead of hardcoding "us", applied
consistently across the three spots that had this pattern.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Timestamp resolution overflow with pandas>=3.0

3 participants